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ABSTRACT 

In an exploration of customizing the Major Field 
Achievement Test in Political Science (MFATPS) , developed by the 
Educational Testing Service, a questionnaire was sent to the heads of 
political science and joint departments which offer a bachelor* s 
degree in Political Science. A total of 1,017 departments were 
surveyed in January 1990; 293 department heads (29%) responded. The 
three alternative customization approaches presented were: (1) use of 
the present MFATPS; (2) the current test plus three optional modules 
in subareas of united states politics; and (3) a revised 
multiple-choice test plus four supplemental modules. Respondents were 
asked to state which alternative was preferable, how much use could 
be expected, and what developmental worfc mig.it be required. Results 
do not clearly support the need for or attractiveness of either of 
the customization models, suggesting retention of the present test. 
Eighteen tables illustrate the sample and responses. Twelve 
appendices provide supplemental information, largely concerning the 
conduct of the survey. (SLu) 
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FINAL REPORT 



SURVEY OF POLITICAL SCIENCE DEPARTMENT CHAIRS ON 

EXPLORATION OF CUSTOMIZATION OF THE 
MAJOR FIELD ACHIEVEMENT TEST IN POLITICAL SCIENCE 

Richard L. Burns 

Senior Development Leader 

May, 1990 

INTRODUCTION 

The proposal approved by the Fund for the Improvement of Post Secondary Education called for the 
"exploration of the customization of the Major Field Achievement Test in Political Science". The 
project planned to explore three issues: User Needs; Technical Issues; and, Practical/Operational 
Issues. The ffrst of these was to be considered through mini-conferences (one was held at the 
Southern Political Science Association meeting in Memphis) and t survey, built upon input from the 
mini-conferences, of department chairs. The carrying out of the other two aspects of the project 
was dependent on the results of the survey. Only "if the results of the meetings and the survey are 
positive in support of one of the models" would ETS move to implement the model on an 
experimental basis. 

Because of limitations in the extent of pretest populations and, hence, items available for use in 
alternative approaches and time constraints, it was necessary to modify the proposal somewhat. 
This was done through discussions with FIPSE staff and documented in a revised plan dated October 
24, 1989. That plan reduced the number of mini-conferences from four to two (only one was 
actually held due to limited participation by attendees at the Southern Association meeting) and the 
limited the alternatives that could be considered. Other aspects of the original proposal remained 
in place. 

Three Alternatives Customization approaches were discussed in the mini-conference. They were: 

1. Current Ma tor Field Political Science Test with Optional Modules Available. 

The available item bank would be used to create three 30- minute modules in subareas of 
United States politics. Institutions could add one or two additional modules to the current 
Major Field Achievement Test for an additional fee and receive the current score reports plus 
scores on whatever modules were also administered. 

2. A Revised Maior Field Test in Political Science to be a United States Politics Test with 
Choice of Additional Mndutes 

The available items would be used in combination with the existing Major Field Achievement 
Test in Political Science to create a 90- minute, 90-item multiple-choice major field test in 
United States Politics. Four additional modules, covering non-U.S. politics material, would 
be created from the existing Political Science test and offered as a supplement to the U.S. 
Politics test. 

3. Publish the Available Item Bank in a Bo. klet That Faculty Could Use to Create Qntional 
Modules of Their Choice. 

Institutions would use the published items to select and print up to 50 items in a module 
which could be given with the existing Major Field Achievement Test in Political Science. 
In addition to scores on the Major Field test, item analysis results would be reported on the 
module items. 



The mini-conference participants had a decided preference for the present test over any of the 
alternatives. However, the last of these alternatives was so thoroughly rejected in the mini- 
conference as to be excluded from consideration in drafting the survey. See Appendix A for a 
summary of the mini-conference and a list of participants. 

A questionnaire was drafted that presented the other two alternatives plus rtte present Major Field 
Achievement Test in Political Science for consideration. It also asked background questions and 
questions ibout strengths and weaknesses of each alternative. The questionnaire was reviewed by 
ETS staff, Dr. Shelia Mann of the American Political Science Association and the FIPSE Contract 
Officer, Dr. Constance Cook. It was field tested with four department chairs at different types of 
institutions. A final version of the questionnaire was finally established and printed. See Appendix 
B for a copy of the questionnaire that was sent 

In January, 1990 the questionnaire was mailed, along with a cover letter from the Executive Director 
of the APSA, Dr. Catherine £. Rudder, to the heads of Political Science departments and joint 
departments offering a bachelors in Political Science. (See Appendix C for a copy of the cover 
letter.) A total of 1,017 departments were surveyed. A follow-up mailing was made to all 
departments that had not responded by the initial deadline of January 22. Data were analyzed from 
293 departments or 29% of the total population. Appendix D lists the departments responding by 
APSA type classification. 

The key question the survey sought to answer is: 

whether one or both customization alternatives would be more attractive to users than the 
existing, non-customized Major Field Test in Political Science? If not, that would raise a 
serious question as to the desirability of developing the customized version? 

A second question is: 

which of the two customization alternatives is most attractive to users. 

A third question is: 

how much actuai use might be expected for each of the customized alternatives? Would this 
use be sufficient to support the technical analysis necessary to make the customization 
professionally acceptable, and keep the cost of operating the customized options realistic? 

Finally. 

what information does the survey provide that can help shape any further developmental 
work? 

This report presents a description of the sample who returned questionnaires. It then addresses the 
key questions raised by the questionnaire. Finally, information about detailed reactions to the three 
alternatives is presented. The report closes with a set of conclusions. 

SAMPLE RESPONDING 

The comparison of the sample to the population in terms of types of departments is shown in Exhibit 
1. A chi-square analysis indicates that significant differences exist between the two distributions. 
Hence, in terms of type of institution, the sample is not representative of the total population; it is 
weighted toward institutions that award a Ph.D. or MA. degree and under-represented by joint 
departments awarding BA degrees. The results from the sample cannot be directly applied to the 
population in terms of projections. In addition, v * cannot assume that those who did not return the 
questionnaire would have responded in a fashion similar to those who did, e.g., the non-respondents 
may well have lower interest in the whole area of assessment, or a major field test, and did not 
bother to return th» questionnaire. 
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Exhibit 2 shows • distribution of Dumber of graduates per departments by total and type of 
department Larger departments (26 or more graduates per year) made up almost half of the 
departments responding (47%). As would be expected the Ph.D. and Master's departments tend to 
be larger. 

No department offers all the specialties available; over half offer American Government nd 
International Relations. A significant percentage, 32%, offer no specialization. (See Exhibit 3.) 
Specialties other than those listed that are cTfered by departments are listed in Appendix E. 

A bachelors degree was the highest degree awarded in 60% of the institutions responding. The. * 
was some inconsistency in reporting, e.g., Ph.D. department: indicating the bachelors degree as the 
highest degree awarded, suggesting either some change in the nature of t**e department or a 
misunderstanding of the question. (See Exhibit 4.) 

Slightly less than 30% of the departments report currently being involved in formal assessment of 
majors in Political Science. The percentages are fairly consistent across types of institutions. (See 
Exhibit 3.) 

Of those currently doing assessment, the Senior Thesis and/or Project was the most common type 
of assessment (42%) followed by Other (29%) and the Major Field Achievement Test in Political 
Science (24%). There were some sharp differences by type of institution; no Ph.D. granting 
institutions used the Major Field test and only 17% used Senior Thesif /Projects, while 40% of the 
Master's Degree departments used the Major Field Test and 20% Senior Thesis/Project For separate 
departments with bachelors degree highest, 26% used the Major Field test while 61% used a Senior 
Thesis/Project. The most frequently cited "other* assessment methods was a senior seminar/alumni 
questionnaire (6). (See Exhibit 6 and, for a full listing of other assessment methods identified, 
Appendix F.) 

Less than a third of the departments not now engaged in assessment feel they will be undertaking 
such assessment within the next two years. However, almost as many departments are uncertain 
about assessment plans (29% vs 27%). More Ph.D. and Masters departments anticipate moving to 
assessment than bachelors departments. (See Exhibit 7.) 

ARE THE CUSTOMIZATION ALTERNATIVES MORE ATTRACTIVE? 

Alternative 1— the current Major Field Achievement Test in Political Science— is clearly more 
attractive and more popular than either of the "customized" alternatives, both when considered 
independently, i.e., only in terms of the particular alternative, and comparative. 

When asked to give a reaction to each alternative, the responses from all institutions responding 
was: 

Alternative Positive Negative 

(Excellent and Very Good) (Very Poor and Poor) 

Current Test 48% 22% 

Optional Additional Module 27% 31% 
U.S.Politics Focus w/ 

Additional Modules 24% 37% 

These preferences, with some differences in degree, held when the data were looked at by type of 
institution. No institutional group preferred either customization option over the current test (See 
Exhibit 8.) 



In terms of likelihood of use of etch ll tentative, the current test would be much more widely used 
than either customised option. The responses from til institutions responding to how likely they 
would be to use the alternative were: 

Alternative Definitely Use/ Probably Not/ 

Probably Use Definitely Not Use 

Current Test 33% 34% 

Optional Additional Module 15% 48% 

U^.PoIitics Focus w/ 

Additional Modules 15% 54% 

Again, the pattern held for all types of institutions. (See Exhibit 9.) 

Viewed on a comparative basis, i.e, when asked to rate one alternative in relation to the other two, 
over twice as many institutions chose the current test first over etch of the alternatives (131 vs 57 
and 43). When asked if they needed to assess their majors which alternative they would use, 34% 
would use the current alternative vs. 17% alternative 2 and 8% alternative 3. Results by type of 
institution were similar. (See Exhibit 10.) 



WHICH OF THE TWO CUSTOMIZATION MODULES IS MOST ATTRACTIVE? 

In comparing the two alternatives, overall the respondents showed considerably more interest in the 
cpiion&l additional module than in the restructured VS. Politics-Focused test with additional VS. 
Politics modules. However, there were some differences by type of department If continued 
consideration were to be given to any customization alternative, additional optional modules would 
be the alternative of choice. (See Exhibits 8-10.) 

WHAT ARE THE PERCEIVED STRENGTHS AND WEAKNESSES OF THE ALTERNATIVES? 

Survey respondents were asked, for each alternative, to identify the principle advantages and 
disadvantages of the alternative. The principle advantages of the current Major Field test in Political 
Science were the content coverage (28%), the availability of full comparative data (21%) and the time 
required ( 1 7%). These continued to be the three most frequently identified advantages for Ph.D. and 
Master's departments, with different percentages, but for bachelor's departments and joint 
department's availability of the 50- locally written questions replaced the time required in the top 
three. Only a few responded to the "Other* option. (See Exhibit 1 1 and Appendix C.) 

The principle limitations identified for the Major Field alternative were, cost (26%), content coverage 
(22%), lack of additional content modules (12%), and other (12%). The most frequently identified 
limitation other was multiple-choice test/not essay examination (17 departments). These were the 
choices across types of departments as well. (See Exhibit 12 and Appendix H.) 

When asked to identify what they would like to see changed or added to the current Major Field 
test, the most frequent responses were: reduce time/cost (9 departments), needs essay questions (6 
departments), more international relations questions (6 departments), and ottar tests, e.g. public 
administration (5 departments). (See App, >dh I.) 

For the additional optional modules alternative, additional content coverage (29%) and the choice 
of modules (22%) were the most frequently chosen advantages. These were the most popular 
advantages across types of departments as well with some differences in the percentage choosing 
e»?h advantage. (See Exhibit 11 and Appendix C.) 
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The most frequently cited limitations for the additional optional modules were, cost (19%), choice 
of modules (17%), other (13%), and extent of modules (10%). These were the same for the different 
types of departments with differences in ordering of the limitations. The most frequently cited 
"other* limitations were multiple choice format (9 departments) and focus on U. S. politics (7 
departments). (See Exhibit 12 and Appendix H.) 

Departments were asked which of the additional modules that would be made available they would 
most likely use. Institutions, policies and process would be the most frequently selected (41%) 
followed by constitutional law/judicial process (37%), and none of these (22%). This order held 
across the different types of departments. (See Exhibit 13.) 

The most frequently identified change or addition to the additional optional modules alternative 
were: modules with a comparative politics/international relations emphasis (8 departments); and, 
public administration (8 departments). (See Appendix I.) 

The most frequently chosen advantage for the VS. Politics Focus with additional modules were 
coverage of modules (17%), choice of modules (16%), and focus on U. S. Politics (13%). These were 
the most popular advantages across types of departments with some differences in percentage 
choosing each advantage. (See Exhibit 1 1 and Appendix G.) 

The focus on U. S. politics (225) and cost (17%) were the most frequently identified limitations for 
the U. S. Politics focus with additional modules. These were consistent across types of departments. 
(See Exhibit 12 and Appendix H.) 

Departments were asked which of the additional modules available under the US. politics focus 
alternative they would choose. The most popular options were: international relations (51%); 
comparative politics (45%); and, Political Theory and History (37%). These were the top three across 
types of departments. (See Exhibit 14.) 

Adding a module on public administration (6 departments) and adding essay questions (5 
departments) were the most cited additions or changes to the U. S. politics focus alternative. (See 
Appendix I.) 

Respondents were asked an open ended question on what other suggestions they had for improving 
major field assessment in Political Science. The three most frequently identified were: need or add 
essay (9 departments); offer option of locally generated add-ons (5 departments)— interestingly that 
option is now available with the current test, cs indicated in the description of alternative 1; reduce 
cost (4 departments). (See Appendix 1 for full list of suggestions made.) 

WHAT LEVEL OF USE COULD BE EXPECTED FOR CUSTOMIZATION MODULES? 

If the most popular of the customization alternatives (Optional Modules Available) were to be 
developed in spite of the limited positive response, how much use would it be likely to attract? 
Some feel for this question can be obtained by relating several of the questions raised in the survey 
to one another. If the percentage choosing each alternative to the forced-choice question of which 
assessment devise the institution would choose if thev needed to undertake ma tor field assessment 
is applied to the number of institutions in the response sample we see that 95 institutions would use 
the current test, 45 would use alternative 2, and only 24 would use alternative 3. By relating these 
results to the question on size of department, the projected number of students who would be tested 
under these conditions would be 3,377 for the current test, 1,495 for the Optional Module alternative 
and 628 for the VS. Politics- Focused alternative, all were available. (See Exhibits 10 and 14 and, for 
a listing of the "other" responses, see Appendix K.) 
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However, both these numbers and the higher numbers tint would be generated by applying the 
percentage to the number of graduates from all departments, are unrealistictlly high. They represent 
the highest volumes possible under the moat favorable conditions not what the volumes are lihelv. 
to be in the short term. If only the departments currently engaged in formal assessment of the major 
are used, the projections change dramatically to 788 for the current test, 310 for the Additional 
Optional Module approach and 58 for the US. Politics-Focus alternative. (See Exhibits 16 and 17.) 

Even these data are inflated, since the number indicating use of the current test (788) exceeds by 127 
(about 20%) the number tested in all institutions in past yean. The percentage increase that would 
result from creating the additional optional module can be applied against volume experience with 
the present Major Field Achievement Test in Political Science, and more realistic projections 
obtained. The volume for the present test over the past two years has averaged 661 annually; far 
below the projected volume generated from analysis of the survey data. By taking the percentage 
increase reflected for the Additional Optional Module alternative and applying it to this figure we 
obtain the more realistic volume estimate of 290 students tested for the Optional Modules approach. 

This number, even if inflated by 10% to 15% to allow for normal program growth, could not support 
the technical work required to make the customization alternative professionally acceptable because 
the volume would be spread over the three optional modules. The survey results indicate that the 
volume would not be spread evenly across the three modules. Rather, 1 19 would choose Institutions, 
Policies, and Process, 35 Political Behavior and 107 Constitutional Law/ Judicial Process. Normally 
a minimum of at least 250 students per module are required for minimal statistical work on the 
module in order to report a score. 

Respondents were ask, about each alternative, why they responded as they did to the question of 
likelihood of use. Responses have been complied and are presented in Appendix L by positive, 
neutral and negative responses to the use question. The most frequent reasons given by those 
responding positively to the current Major Field alternative were "content coverage" (14) and 
"comparative data" (9). The most frequent reason given by those responding negatively were, "not 
appropriate for the curriculum/content coverage problems" (21) and "will cot use multiple-choice 
exam/problems with standardized exams" (24). With regard to the additional optional module 
alternative, the most frequent responses of those responding positively were: "reflects curriculum 
better/coverage" (5) and "flexibility* (4). The most frequent responses of those responding negatively 
were: "too focused on U.S. Politics" (17) and "multiple choice/standardized test" (17). Those 
responding positively to the U.S. Politics focused test with additional modules save "focus on U.S. 
Politics" (6) most frequently as the reason why. Of those responding negatively to this alternative, 
"too focused on U.S. Politics/too limited and/or narrow" (47) and "multiple choice" (10) were the most 
frequently given reason. 

SHOULD A CUSTOMIZED POLITICAL SCIENCE ALTERNATIVE BE DEVELOPED? 

Based on these results, the need for either customized version is not supported. If the assumption 
that customization is responsive to users (department) needs was to be upheld, one of the 
customization modules should have been more attractive than the non-customized test and should 
project more use than the non-customized alternative. Apparently the bulk of the institutions are 
satisfied, indeed, feel better served by the current Major Field Achievement Test in Political Science 
than by either of the proposed alternatives. While some additional users might be attracted to a 
customized alternative, they would be insufficient, in the short term, to support either the 
operational cost or the technical needs of the customized alternative. 
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It may be that if some other, more extensive form of customization had been proposed the results 
might have been different. This, however, was not possible within the constraints of the current 
project nor is it practical in terms of the cost that would be involved in the operation of the 
customized program. 

Thus, the survey data do not support the creation of a a "omization alternative in Political Sci nee. 
While the items and technology exist to create a customized test, it is unlikely that enough users 
would be available in the first years of use to permit the necessary statistical work to support the 
reporting of scores on the module. In addition, the volume that would be generated would not 
justify the cost of the development or its operational expenses. 

CONCLUSION 

Since the survey results clearly do not support the need for or the attractiveness of either of the 
customization models, and since the number of cases that would likely result if the customization 
were created would not support the technical needs of the model, ETS does not feel the project 
should be carried forward to experimental implementation. 

While the outcome of the project is disappointing to both ETS and FIPSE, some valuable knowledge 
was gained. 

1. The cost and difficulty of creating a professional quality item bank, at least in Biology and 
Political Science for major field assessment purposes are too great to be practical, given 
current levels of interest of college and university faculty. 

2. In spite of frequently heard and logically reasonable appeals for major field assessment 
measures that can be customized to meet departmental needs, at least in Political Science, 
the limited customization approaches considered are not as attractive as a uniform approach. 

3. If departments of political science are forced to undertake major field assessment, a un: form, 
standardized test would be the most popular alternative. 

4. Without a specific requirement for major field assessment, the number of departments and 
number of students using a uniform, standardized test remains relatively small in comparison 
to its potential. 
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Exhibit 1 



Type of institution 

Ph.D. in Pol. Sci. 
M.A. in Pol. Sci. 
Sept. Dept. B.A. 
Joint Dept. B.A. 
Total 



Comparison of Sample with Population 

Sample Population 
N % N % 



45 
59 

137 
52 

293 



15% 
20% 
47% 
18% 
100% 



124 
155 
476 
262 
1,017 



12% 
15% 
47% 
26% 
100% 
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Exhibit 2 

DISTRIBUTION OF DEPARTMENTS BY NUMBER OF GRADUATES IN 1988-89 



No. Gradu^es 


Total 


Type A 


Type B 


Type C 


Type D 




N - 


293 


N 


- 45 


N - 59 


N 


- 137 


N 


- 52 




N 


% 


N 


% 


N 


% 


N 


% 


N 


% 


No Response 


18 


6% 


0 


0% 


2 


3% 


9 


7'4 


7 


14% 


None 


4 


1% 


1 


2% 


0 


0% 


2 


2% 


1 


2% 


1- 5 


23 


8% 


0 


0% 


3 


5v 


9 


7% 


11 


21% 


6-10 


37 


13% 


0 


0% 


2 


3% 


20 


15% 


15 


29% 


11-15 


32 


11% 


1 


2% 


3 


5% 


19 


14% 


9 


17% 


16-20 


18 


6% 


0 


0% 


4 


7% 


11 


8% 


3 


6% 


21-25 


23 


8% 


0 


0% 


5 


9% 


16 


12% 


2 


4% 


25-50 


51 


17% 


5 


11% 


14 


24% 


28 


20% 


4 


8% 


51-100 


32 


11% 


6 


13% 


13 


22% 


13 


10% 


0 


0% 


< 100 


55 


19% 


32 


71% 


13 


22% 


10 


7% 


0 


0% 


Mean 


42 grad. 


86 


Grad. 


54 


Grad. 


30 


Grad. 


9 


Grad. 



1 f) 




Exhibit 3 

DISTRIBUTION OF DEPARTMENTS BY AREAS OF SPECIALIZATION OFFERED 



Area of Special. 


Total 


Type A 


Type 8 


Type C 


Type D 




N « 


293 


N ™ 


$5 


M 

N m 


59 


N ■» 


137 


mm 


K A 

52 




N 


% 


N 


% 


mm 

N 


a 

% 


mm 

N 


B 


N 


MX 
t 


Aaer. Govt. 


161 


551 


Zo 


.1)4 

62% 


36 


61% 


77 


c a. 

56 - 


A A 

20 


aaa 

39% 


International Rel. 


161 


55% 


30 


67% 


38 


64% 


77 


56% 


16 


31% 


Comparative Pol. 


131 


45% 


30 


67% 


31 


53% 


60 


44% 


10 


19% 


Political Theory 


103 


35% 


22 


49% 


29 


49% 


45 


33% 


7 


14% 


Public Admin. 


88 


30% 


15 


33% 


29 


49% 


32 


23% 


12 


23% 


Public Lav 


80 


27% 


14 


31% 


27 


45% 


30 


22% 


9 


17% 


Public Policy 


47 


16% 


15 


33% 


13 


22% 


15 


11% 


4 


8% 


Quant. /Res. Meth. 


15 


5% 


7 


16% 


4 


7% 


2 


2% 


2 


4% 


Other (1) 


24 


6% 


5 


11% 


4 


7% 


11 


8% 


4 


3% 


Other (2) 


6 


2% 


2 


4% 


1 


2% 


1 


1% 


2 


4% 


Do not Special. 


95 


32% 


13 


29% 


16 


27% 


46 


34% 


20 


39% 



17 



Exhibit 4 

DISTRIBUTION OF DEPARTMENTS BY HIGHEST DEGREE AWARDED 



Area of Special. 


Total 




Type A 




N - 293 


N - 


45 




N 


% 


N 


% 


Bachelors Degree 


176 


60% 


2 


4% 


Masters Degree 


57 


20% 


0 


0% 


Ph.D. Degree 


45 


15% 


43 


96% 


No Response 


15 


5% 


0 


0% 



19 




Type B Type C Type D 



N - 


59 


N - 


137 


N - 


52 


N 


% 


N 


% 


N 


% 


3 


6% 


128 


93% 


43 


83% 


54 


92% 


1 


1% 


2 


4% 


1 


2% 


1 


1% 


0 


0% 


1 


2% 


7 


5% 


7 


14% 




Exhibit 5 

DISTRIBUTION OF DEPARTMENTS BY CURRENTLY 
ENGAGEMENT IN FORMAL ASSESSMENT 



Engagement 



Total 



Type A 



Type B 



Type C 



Type D 





N - 


293 


N - 


45 


R - 


59 


N - 


137 


N - 




N 


% 


H 


% 


N 


% 


N 


% 


N 


Yes 


85 


29% 


12 


27% 


15 


25% 


43 


31% 


15 


No 


189 


65% 


33 


73% 


40 


68% 


88 


64% 


28 


No Response 


19 


7% 


0 


9% 


4 


7% 


6 


4% 


9 



29% 
54% 
17% 



21 
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Exhibit 6 

DISTRIBUTION OF DEPARTMENTS ASSESSING BY TYPE OF ASSESSMENT 



Type Assessment Total Type A 





N - 


62 


N - 


4 






% 


N 


% 


Dept. M-C Exam 


5 


6% 


0 


0% 


Dept. Essey Exam. 


13 


15% 


2 


17% 


Dept. tf-C/Essay Ex. 


3 


4% 


0 


0% 


MEAT Political S. 


20 


24% 


0 


0% 


GRE Political S. 


3 


4% 


0 


0% 


Sr. Thesis/Project 


36 


42% 


2 


17% 


Other 


25 


29% 


7 


58% 



?3 



Type B Type C Type D 



N - 


10 


N - 


31 


N - 


13 


N 


% 


'A 


% 


N 


% 


1 


7% 


2 


5% 


2 


13% 


1 


7% 


6 


19% 


2 


13% 


0 


0% 


0 


0% 


3 


20% 


6 


40% 


11 


26% 


3 


20% 


0 


0% 


3 


7% 


0 


0% 


3 


20% 


26 


61% 


5 


33% 


6 


40% 


8 


19% 


4 


27% 
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Exhibit 7 

DISTRIBUTION OP DEPARTMENTS ASSESSING BY INTENTION 
TO BEGIN ASSESSMENT WITH' ' TWO TEARS 



Begin Assessment 


Total 


Type A 




N - 


231 


N - 


41 




N 


% 


N 


% 


Yes 


86 


29% 


15 


33% 


Uncertain 


78 


27% 


9 


20% 


No 


67 


23% 


17 


38% 


No Response 


62 


21% 








Type B 


Type C 


Type D 


N - 49 


N - 


106 


N - 40 


N % 


N 


% 


N % 


22 37% 


36 


26% 


13 25% 


21 36% 


35 


26% 


13 25% 


6 10% 


35 


26% 


9 17% 



• 



Exhibit 8 

Independent Comparison of Alternatives By Total and Type of Institution 

Overall Reaction to Alternatives 



Rating 


Total 




Type A 




Type B 




Type 


C 




Type D 






N - 


293 




N - 


45 




N - 


59 




N - 


137 




N - 


52 






Alternative 


Alternative 


Alternative 


Alternative 


Alternative 




1 


2 


3 


1 


2 


3 


1 


2 


3 


1 


2 


3 


1 


2 


3 


Excel. 


9% 


5% 


4% 


0% 


2% 


0% 


12% 


3% 


2% 


7% 


5% 


6% 


17% 


8% 


6% 


V.Good 


39% 


22% 


20% 


24% 


16% 


9% 


46% 


20% 


27% 


41% 


25% 


18% 


37% 


21% 


25% 


Neutral 


28% 


37% 


31% 


29% 


24% 


31% 


25% 


39% 


34% 


26% 


39% 


29% 


35% 


37% 


33% 


Poor 


16% 


22% 


26% 


24% 


33% 


33% 


14% 


27% 


24% 


20% 


18% 


28% 


2% 


14% 


19% 


V.Poor 


6% 


9% 


11% 


13% 


16% 


20% 


3% 


7% 


5% 


6% 


8% 


14% 


4% 


6% 


4% 


No Reap. 


3% 


7% 


8% 


9% 


9% 


7% 


0% 


3% 


9% 


1% 


4% 


6% 


6% 


15% 


14% 



P. 7 



Exhibit 9 



Independent Comparison of Alternatives By Total and Type of Institution 

Likelihood of Dae 





Total 




Type A 




Type B 




Type C 




Type D 






N - 


293 




N - 


45 




N - 


59 




H - 


137 




N - 


52 




Likellh. 


Alternative 


Alternative 


Alternative 


Alternative 


Alternative 




i 


2 


3 


1 


2 


3 


1 


2 


3 


1 


2 


3 


1 


2 


3 


Definitely 


6% 


1% 


1% 


0% 


0% 


0% 


10% 


0% 


0% 


5% 


0% 


2% 


8% 


6% 


0% 


Probably 


27% 


14% 


14% 


13% 


11% 


4% 


36% 


10% 


20% 


29% 


17% 


14% 


27% 


12% 


15% 


Hlght/MNot 


31% 


31% 


25% 


29% 


24% 


24% 


29% 


34% 


27% 


30% 


32% 


22% 


37% 


31% 


31% 


Prob. Hot 


21% 


31% 


37% 


29% 


31% 


38% 


15% 


34% 


37% 


23% 


34% 


38% 


14% 


23% 


31% 


Defln.Hot 


13% 


17% 


17% 


24% 


27% 


31% 


10% 


15% 


5% 


12% 


16% 


20% 


12% 


15% 


10% 


Ho Reap. 


2% 


6% 


7% 


4% 


7% 


2% 


0% 


7% 


10% 


2% 


2% 


4% 


4% 


14% 


14% 



PS 




Exhibit 10 



Direct Comparison of Alternatives 
Total N - 293 

Number and Percentage choosing each Alternative First by Total and Type 



Alternatives 


Total 




Type 


A 


Type B 


Type C 


Type 


D 






N 


% 


N 


% 


N 


% 


N 


% 


N 


% 


Alternative 


1 


131 


>i% 


16 


36% 


33 


56% 


60 


44% 


22 


42% 


Alternative 


2 


57 


20% 


8 


18% . 


8 


14% 


32 


23% 


9 


17% 


Alternative 


3 


A3 


15% 


8 


18t* 


10 


17% 


18 


13% 


7 


14% 


Mo Response 




62 


21% 


13 


29% 


8 


14% 


27 


20% 


14 


27% 



Selection of Alternative by Total and Type Institution 





Total 


Type A 


Type B 


Type C 


Type D 


Alt 1 


34% 


18% 


44% 


36% 


35% 


Alt 2 


17% 


7% 


14% 


20% 


19% 


Alt 3 


8% 


4% 


fe* 


7% 


13% 


Own Test 


16% 


16% 


19% 


18% 


8% 


Other 


10% 


31% 


9% 


6% 


6% 


No Response 


15% 


24% 


8% 


13% 


19% 



30 
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Exhibit XI 

Independent Comparison of Alternatives By Type of Institution 

Principle Advantage 
Percent Selecting Each Advantage 





Total 




Type A 




Type B 




Type 


C 




Type D 






N - 


293 




N - 


43 




N - 


59 




N - 


137 




N - 


52 




Advantage 


Alternative 


Alternative 


Alternative 


Alternative 


Alternative 


1 


2 


3 


1 


2 


3 


1 


2 


3 


1 


2 


3 


1 


2 


3 


TlmeReq . 


17% 


9% 


9% 


11% 


11% 


9% 


7% 


5% 


4% 


4% 


2% 


2% 


6% 


4% 


4% 


Coat 


3% 


2% 


1% 


4% 


0% 


2% 


2% 


5% 


0% 


2% 


1% 


2% 


4% 


2% 


0% 


Content Gov. 


28% 






13% 






34% 






26% 






39% 






AddContCov. 




29% 






27% 






29% 






29% 






35% 




Cov.ofMod, 


m m 




17% 






18% 






19% 






18% 






12% 


ChoiceNod . 




22% 


16% 




13% 


18% 




19% 


24% 




30% 


15% 




14% 


15% 


AddlteaOpt. 


11% 






9% 






9% 


m m 




12% 






10% 






Focus JSPol 






13% 






11% 






3% 






15% 






29% 


FullCompD. 


21% 






22% 






29% 






20% 






12% 






Other 


3% 


5% 


5% 


2% 


4% 


2% 


0% 


3% 


3% 


5% 


6% 


71 


4% 


6% 


4% 


Ho Response 


28% 


37% 


41% 


38% 


44% 


47% 


17% 


36% 


41% 


31% 


34% 


41% 


27% 


40% 


39% 
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Exhibit 12 



D Is Advent. 



TiaeReq. 
Cost 

Content Cov. 

LackAddMod. 

ExtKodules 

ChoiceKod. 

CovKodules 

FocusUSPol . 

LiaCompData 

LacfcCompData 

Other 

NoResponse 



Independent Comparison of Alternatives by Type of Institution 

Principle Limitation 
Percent Selecting Each Limitation 



Total 
N - 293 

Alternative 

12 3 

3% 3% 3% 

26% 19% 17% 
22% 
12% 

-- 10% 
17% 

-- 8% 

-- 22% 

6% — 
9% 

12% 13% 7% 

25% 32% 35% 



Type A 
N - 45 

Alternative 
12 3 

4% 4% 2% 
18% 13% 16% 

20% -- 
11% — — 
2% 

13% -- 
4% 

— 29% 
2% 

2% 

20% 7% 

44% 40% 



18% 
29% 



Type B 

N - 59 

Alternative 
12 3 

3% 2% 2% 

37% 27% 22% 

24% -- 

12t -- 

-- 12% -- 

— 20% -- 

— — 12% 
-- 22% 

7% -- 
9% 

7% 14% 3% 
17% 19% 31% 



Type C 

N - 137 

Alternative 
12 3 

4% 4% 3% 
25% 16% 15% 
23% 
12% 

11% 

-- 20% - - 
-- 6% 
-- 22% 
7% -- 
11% 
10% 7% 
33% 36% 



Tyje 0 

N - 52 

Alternative 
12 3 

0% 2% 4% 



19% 
14% 



10% 
10% 



8% 



11% 
26% 



10% 
17% 

- 12% 
15% 17% 8% 
27% 33% 31% 



33 



ERIC 




Exhibit 13 

DISTRIBUTION OF INSTITUTIONS BT WHICH ADDITIONAL MODULES WOULD BE USED 
OPTIONAL MODULES AVAILABLE ALTERNATIVE (2) 



Additional Modules Total Type A 





N - 


231 


N - 


41 




N 


% 


N 


% 


In* tit. Pol. & Proc. 


121 


411 


21 


47% 


Political Behavior 


34 


12% 


6 


13% 


Const. Lav/Jud. 










Proc . 


109 


37% 


10 


22% 


None of These 


64 


22% 


9 


20% 



35 




Type B 


Type C 


Type D 


N - 49 


N - 


106 


N - 40 


H % 


N 


% 


N % 


19 32% 


63 


45% 


18 35% 


11 19% 


10 


7% 


7 14% 


16 27% 


62 


45% 


21 40% 


16 27% 


30 


22% 


9 17% 
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Exhibit 14 

DISTRIBUTION OF INSTITUTIONS BY WHICH ADDITIONAL MODULES WOULD BE USED 

U.S. POLITICS FOCUS ALTERNATIVE (2) 



Additional Modules 


Total 


Type A 




N - 


231 


N - 


41 




N 


% 


N 


% 


Comparative Pol. 


132 


45% 


19 


42% 


Pol. Theory & Hist. 


107 


37% 


15 


33% 


Internetl. Rel. 


148 


51% 


18 


40% 


Methodology 


30 


10% 


4 


9% 


None 


37 


13% 


7 


16% 



rt7 




Type Z Type C Type D 



N - 


49 


N - 


106 


N - 


40 


N 


% 


N 


% 


N 


% 


31 


53% 


63 


46% 


19 


37% 


26 


44% 


52 


38% 


14 


27% 


33 


56% 


75 


55% 


22 


62T» 


12 


20% 


11 


8% 


3 


6% 


7 


12% 


14 


10% 


9 


17% 



3B 




Exhibit 15 

Projected Volume of Sample Based on Preference Among alternatives If 
Assessing Major-Institutions and Students 



Alternative 



Alternative 1 
Alternative 2 
Alternative 3 
Other 



Projected Volume 
Institutions Students 



95 

45 
24 
73 



3,377 
1,495 
628 
3,848 
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Exhibit 16 



EXTENT OF USE OF ASSESSMENT APPROACH OF THOSE CURRENTLY 

ENCAGED IN ASSESSMENT 



Total Type A Type B Type C Type D 





N - 


(15 


N - 


12 


N - 


15 


N - 


43 


N - 


15 




N 


% 


N 


% 


N 


% 


N 


% 


N 


% 


Dept. M-C Exam 


5 


6% 


0 


0% 


1 


7% 


2 


5% 


2 


13% 


Dept. Essay 


13 


15% 


2 


17% 


1 


7% 


8 


19% 


2 


13% 


Dept. M-C/Essay 


3 


4% 


0 


0% 


0 


0% 


0 


0% 


3 


20% 


XFAT Political S. 


20 


24% 


0 


0% 


6 


40% 


11 


26% 


3 


20% 


ORE Subject Test 


3 


4% 


0 


0% 


0 


0% 


3 


7% 


0 


0% 


Senior Thesis/ 






















Project 


36 


42% 


2 


17% 


3 


20% 


2C 


61% 


5 


33% 


Other 


25 


29% 


7 


58% 


6 


40% 


8 


19% 


4 


27% 



40 
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Exhibit 17 



EXTENT OF PROJECTED TOTAL FUTURE ASSESSMENT ACTIVITY 



PLANNED FUTURE 
ASSESSMENT 



Flan Assessment N/I 
Next Two Tears 
Uncertain 
No Flans 



Total 
N - 186 
N % 



2296 27% 
2774 33% 
3290 39% 



Type A 
N - 12 
N % 



776 27% 
1590 54% 
552 19% 



Type B 
N - 15 



732 31% 
1140 48% 
476 20% 



Type C 
N - 43 
N 1 



504 23% 
500 23% 
1173 54% 



Type D 
N - 15 



52 25% 
106 51% 
51 24% 



42 
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Exhibit 18 



Projected Volume of Sample Based on Preference Among Alternatives If 
Assessing Major-Institutions and Students 
(For those institutions currently engaged in formal assessment) 



Alternative Projected Volume 





Institutions 


Students 


Alternative 1 


30 


788 


Alternative 2 


12 


330 


Alternative 3 


6 


58 


Other 


24 


1.036 



44 
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APPENDIX A 



SUMMARY OF MEETING WITH DEPARTMENT CHAIRS 
IN POLITICAL SCIENCE ON FIPSE PROJECT-EXPLORING 
THE CUSTOMIZATION OF MAJOR FIELD TESTS 



Following a meeting with FIPSE staff on October 20, department chairs scheduled to attend the 
Southern Political Science Association conference were identified. Initially, 40 chairs were 
identified. However, upon calling a number of these were either no longer the chair (in some cases 
the new chair was attending and was identified) or were not planning to attend. Of those contacted, 
11 agreed to meet with on Thursday evening, November 2. None were available for a Friday 
meeting. Of those 11, seven showed up. Those who agreed and those who attended are identified 
on the attached Exhibit A- 1. The bulk of those attending were from public institutions of moderate 
size; the largest were University of Georgia and Old Dominion. Only Old Dominion was a Political 
Science Major Field Achievement Test user. Those agreeing to attend were sent a letter explaining 
the purpose, identifying the time and place of the meeting and a copy of the latest Major Field 
Achievement Test brochure as background. 

While the participants ate, the Project Director, Dr. Richard Burns, made a presentation that covered 
the following: 

| Purpose of the meeting 

I Background on the Assessment Movement in higher education 

| Description of the first two years of the FIPSE Item Bank Project 

| Description and purpose of the current project 

| Description of the Major Field Achievement Testing Program 

| Description of the Major Field test in Political Science 

| Description of the FIPSE Pol* tical Science Item Bank Status 

During and following the presentation questions from the participants were responded to. This 
discussion showed a wide degree of experience with and interest in assessment in the major. Several 
were very interested and current facing the problem. Others had given little thought to it. The most 
negative to the idea throughout the meeting came up to the Project Director as he was leaving and 
said he had learned a great deal which he was likely to need in the future. There was general 
agreement that faculty interest in such assessment was quite low, even where it was being mandated. 
There was also agreement that it wasn't likely to take place without external pressures, i.e., the 
faculty were not going to do it for their own purposes. 

When everyone seemed to be relatively clear on the background, the Project Director presented the 
alternatives individually and raised questions about each before going on to the r.ext The 
alternatives and a summary of the major points raised in reaction to each are summarized below. 



I 



Augmented Optional Model and Student Choice (see Exhibit 2 & 3). 



The overall reaction was generally negative to this concept It was seen as a possible 
way to overcome laziness on the part of faculty but severely limited by the choice 
of options. When asked if going to do assessment of the major they would use none, 
one or two optional modules, ill indicated they would use none of diem. They would 
stick with the basic Major Field test Making additional modules in other areas of 
Political Science would make it more attractive. While some choice was seen as 
positive (even if they wouldn't use it), the small number of modules available and 
their limitation to U.S. Government was seen as a major drawback. Some felt that 
the initial lack of comparative data on the module could be an advantage. Giving the 
students a choice of options was not seen as a strong enhancement to this alternative 
and did not change any views about its attractiveness. There was a general feeling 
that departments would choose the easiest lest costly and most direct approach, i.e., 
the existing test 

| Revised Major Field Test with Alternative Modules (see Exhibit 4). 

This alternative was less popular than the first primarily because of the US. 
Government emphasis that it would, by necessity, carry. There was a general feeling 
that "faculty would revolt" at a test whose emphasis was U.S. Government since many 
already feel there is a bias toward US. Government In addition, the data resulting 
from such a test would be likely to produce biased dam at many institutions since the 
better students do not specialize in VS. Government This option could be made 
attractive if there were enough separate modules so that institutions could structure 
their own test entirely. (Item pool limitations are such that it would not be possible 
to construct such t test.) However, it was recognized that this would suffer from a 
lack of comparative dam. None of those present would choose this alternative, if it 
were available, in lieu of the present Major Field test in Political Science. 

| Department Creation of Optional Modules from Printed Item Pool (see Exhibit 5). 

This alternative was seen as "totally useless*. (The reaction of the Chair's to this 
alternative was very simila to Connie Cook's.) The lack of item security, comparative 
dam and feed-back were seen as major draw-backs. Something similar is already 
available by 14 sub-discipline of Political Science from ARNO Press and it is not 
widely used. Since this approach would again be limited to U.S. Government, it 
would be even less useful. No one would choose to use this alternative if it were 
available. 

The overall judgement of the group was that none of the options were as good as using the current 
Major Field Achievement Test in Political Science. 

It must, of course, be remembered that this represents only the reactions and judgements of an 
available set of a few Department chairs. There is no certainty that a different group would have 
reacted the same. Indeed, their negative reaction to the focus of the item bank and hence the 
alternatives on VS. Government was not shared by the "Design Committee" that formulated the 
Political Science item bank structure. 

Because of the basically negative reaction, not much was learned from the discussion which would 
change the shape of the alternatives. It is not clear that a second or third group would provide more 
useful information. Because of the difficulty in getting another group soon (there are no more 
regional meetings until spring) and reservations about the likely results, moving to carry out the 
survey now seems appropriate. 



Because of the strong reaction from the group to the idea of publishing the item pool and making 
it available for use and the psychometric limitations of such an approach, and to simplify the survey, 
it will focus on three alternatives: 

| The present Major Field Achievement Test in Political Science (for base line 
purposes). 

| Augmented Optional Model 

I Revised Major Field Test with Alternative Modules 



Exhibit A-l 

PARTICIPANTS ATTENDING FIPSE PROJECT DISCUSSION OF 
EXPLORING THE CUSTOMIZATION OF MAJOR FIELD ACHIEVEMENT TEST 

IN POLITICAL SCIENCE 



Dr. Glen T. Broach 
Department Chair 
Winthrop College 
Political Science Department 
Rock Hill, SC 29733 

Professor Clarke E. Cochran * 
Department Chair 
Texas Tech University 
Political Science Department 
Lubbock, TX 79409 

Professor Eldon Eisenach * 
Department Chair 
University of Tulsa 
Political Science Department 
600 South College Avenue 
Tulsa, OK 74104-3189 

Professor Bahrain Farzanegan * 
Department Chair 

University of North Carolina- A she ville 

Political Science 

One University Heights 

Asheville, NC 28804-3299 

Professor Lawrence J. Hanks 
Department Chair 
Tuskegee University 
Political Science Department 
Tuskegee Institute, AL 36088 

Professor Ronald M. Labbe * 
Department Chair 

University of Southwestern Louisiana 
Political Science Department 
Box 4 1652 
Fafayette* LA 70505 



Dr. Thomas P. Lauth * 
Department Chair 
University of Georgia 
Political Science Department 
Baldwin Hall 
Athens, GA 30602 

Professor Connie P. Mauney 
Department Chair 
Emporia State University 
Political Science Department 
Emporia, KS 66801 

Professor Robert H. Swansbrough 
Department Chair 

University of Tennessee - Chattanooga 
Political Science Department 
615 McCallie Avenue 
Chattanooga, TN 37403 

Professor Mary Ann Tetreault * 
Department Chair 
Old Dominion University 
Political Science A Geography 
Norfolk, VA 23508 

Professor Donald T. Wells * 
Department Chair 
West Georgia College 
Political Science Department 
Carrollton, GA 30118 



* Participants who attended the scheduled session on November 2. 
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Exhibit A-2 

MAJOR FIELD ACHIEVEMENT TEST CUSTOMIZATION PROJECT 

POLITICAL SCIENCE-ALTERNATIVE ONE 
AUGMENTED MAJOR FIELD TEST 

I USE FIFTY-ITEM OPTIONAL QUESTION PROVISION 
OF CURRENT TEST. 

I INSTITUTION SELECTS ONE OR TWO OF UP TO 
THREE MODULES TO ADMINISTER TO ALL 
STUDENTS IN ADDITION TO FULL MAJOR FIELD 
TEST OR STUDENT SELECTS 

O CONSTITUTIONAL LAW/JUDICIAL PROCESS 

O POLITICAL BEHAVIOR 

O INSTITUTIONS, POLICIES. AND PROCESSES 
I TEST WOULD NOW BE: 

O 120 MINUTES WITH NO ADDITIONAL MODULES 

O 150 MINUTES WITH ONE ADDITIONAL MODULE 

O 180 MINUTES WITH TWO ADDITIONAL 
I WOULD YIELD: 

O SAME DATA AS REGULAR MAJOR FIELD TEST, 

O ADDITIONAL ASSESSMENT INDICATOR FOR 
EACH MODULE CHOSEN 

I INSTITUTIONS WOULD PAY AN ADDITIONAL FEE 
FOR EACH MODULE USED ON A PER STUDENT 
TESTED BASIS 

I COMPARATIVE DATA WOULD BE LIMITED AT FIRST 



Exhibit A-3 

MAJOR FIELD ACHIEVEMENT TEST CUSTOMIZATION PROJECT 
POLITICAL SCIENCE-ALTERNATIVE ONE-A 



AUGMENTED MAJOR FIELD TEST-STUDENT CHOICE 



I SAME AS ALTERNATIVE ONE, EXCEPT: 



O INSTITUTIONS OFFER ALL MODULES AND 
STUDENTS CHOOSE WHICH OF OPTIONAL 
ADDITIONAL MODULES TO TAKE 



O GROUP INFORMATION WOULD BE OF LESS USE 
UNLESS DEPARTMENT WAS LARGE DUE TO 
THE SMALL SAMPLE SIZES RESULTING FROM 
DIFFERENT MODULES TAKEN BY STUDENT 



O WOULD COST MORE THAT ALTERNATIVE ONE 
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Exhibit A-4 

MAJOR FIELD ACHIEVEMENT TEST CUSTOMIZATION PROJECT 

POLITICAL SCIENCE-ALTERNATIVE TWO 

REVISED MAJOR FIELD TEST WITH 
ALTERNATIVE MODULES 



| CURRENT MAJOR FIELD TEST WOULD BE CHANGED 
TO A U. S. GOVERNMENT POLITICAL SCIENCE TEST 
BY REMOVING ALL BUT U.S. GOVERNMENT ITEMS 
FROM CURRENT TEST AND ADDING U. S. 
GOVERNMENT ITEMS FROM POOL. TEST WOULD 
BE 90 MINUTES LONG. 

| INSTITUTIONS COULD THEN ADD ONE MODULE 
FROM CURRENT MAJOR FIELD TEST AND STAY 
UNDER TWO HOURS OR ADD UP TO TWO 
ADDITIONAL MODULES AND ADDITIONAL TIME. 

| MAJOR FIELD RESULTS WOULD NOT BE 
COMPARABLE WITH EXISTING MAJOR FIELD 
SCORES. 

| CURRENT TEST WOULD ALSO BE OFFERED SO USE 
OF CURRENT TEST WOULD DETERMINE EXTENT OF 
COMPARATIVE DATA 

| COMPARATIVE DATA COULD BE LIMITED 
DEPENDING ON OPTIONS CHOSEN- - 
COMPARATIVE DATA WOULD BE BY MODULE 

I ALL ALTERNATIVES MODULES PRINTED IN TEST 
BOOK AS MODULES 

I COSTS TO INSTITUTIONS WOULD BE HIGHER 



Exhibit A-5 

MAJOR FIELD ACHIEVEMENT TEST CUSTOMIZATION PROJECT 

POLITICAL SCIENCE-ALTERNATIVE THREE 

DEPARTMENT CREATION OF OPTIONAL MODULE 
FROM PRINTED ITEM POOL 



fi THE 200 AVAILABLE ITEMS WOULD BE PUBLISHED 
IN A BOOKLET AND MADE AVAILABLE TO MAJOR 
FIELD ACHIEVEMENT TEST USERS THAT WANTED 
TO CREATE OPTIONAL MODULE(S). 



I INSTITUTIONS WOULD SELECT ITEMS FROM 
BOOKLET AND PRINT UP OPTIONAL MODULE OR 
MODULES FOR ADMINISTRATION WITH CURRENT 
MAJOR FIELD TEST. 

| ETS WOULD SCORE AND REPORT BASIC MAJOR 
FIELD TEST RESULTS PLUS PROVIDE ITEM 
ANALYSIS DATA ON OPTIONAL MATERIAL PLUS 
DISKETTE OF ITEM RESPONSES. 

| NO COMPARATIVE DATA COULD BE PROVIDED FOR 
OPTIONAL MODULES. 

I TESTING TIME WOULD BE INCREASED TO 150 OR 
180 MINUTES. 

| ADDITIONAL COST TO INSTITUTIONS WOULD BE 
MINIMAL. 

| ITEM POOL WOULD BE EXPOSED AND MODULE 
COULD BE TAUGHT TO IF INSTITUTIONS WISHED 
TO DO SO 



Appendix B 
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SURVEY QUESTIONNAIRE 
POLITICAL SCIENCE DEPARTMENT CHAIRS 
FIPSE/ETS EXPLORATION OF CUSTOMIZATION OF 
MAJOR FIELD TEST IN POLITICAL SCIENCE 



Background. Educational Testing Service, aware of the pressure departments are under from states 
and accreditation agencies to assess leaning of students In their majors, developed, with the 
Graduate Record Examinations Board, the Major Field Achievement Tests in 1987-88. The Major 
Field Achievement Tests are shorter, less difficult versions of the GRE Subject Tests. They are also 
more closely related to the undergraduate curriculum and can be administered directly by 
departments In IS fields. In addition to an Individual total score (and for six examinations. 
Individual subscores) most tests also provide group data on the subareas covered by the test. These 
data are called "Assessment Indicators* and are presented In the form of the mean percent correct 
for that subarea for the group of students tested within a department. 

Recognizing that this type of standardized, national test couid not meet the needs of all 
departments, ETS obtained a Fund for the Improvement of Postsecondary Education (FIPSE) grant 
to try to develop Item (question) banks In several fields, which could be made available, via a 
personal computer program, with which departments could construct end-of-major examinations 
that reflected their particular curricula. However, after two years of work to try to develop such 
banks in several fields, including political science, we found that the cost and complexities Involved 
made It lmpossibli to accomplish. With th* encouragement of FIPSE and the American Political 
Science Association, w > are currently trying to determine If the Items developed for the political 
science Item bank cou J met' departments* needs if they were offered In a way that would permit 
some customization of the Major Field Achievement Test in Political Science. 

This survey Is an effort to gather information about the extent of the need for such customization 
and the degree of interest there might be In several alternative ways of structuring It. 

Directions. Please review, one at a time, each of three alternative approaches to major field 
assessment and then respond to questions about each. Background questions, designed to help 
analyze the data, are asked at the end of the questionnaire. 

Please mark the answer that most closely reflects your views as chairman of the political science 
department at your institution or provide a brief response where appropriate. It Is not necessary to 
poll or otherwise consult members of your faculty. When completed, the questionnaire should be 
returned to ETS in the prepaid envelope provided. We would appreciate receiving the completed 
questionnaire by January 22. If you have questions about any aspect of the questionnaire, please 
feel free to call. 

Thank you very much for your cooperation in completing the questionnaire. 



Richard L. Burns 
Senior Development Leader 
Educational Testing Service 
(609-243-8087) 



identification data: 



§ 



Institution Name 



City and State 
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AlKrnaUrt 1. Current Malor Field Test In Politic*! StkflCj; 



In this alternative, the Major Field Test in Political Science, as It currently exists would be available 
for use without further Intervention by ETS. This Is a two-hour teat consisting of 118 multiple- 
choice items that yields total score and Assessment Indicators (see cover sheet) for the group of 
students tested within a department. The Political Science Assessment Indicators are: U.S. 
Government (39); Comparative Political Systems (27); International Relations (IS); Political Theory 
and the History of Political Thought (24); and Methodology (11). (The numbers In parenthesis are 
the number of Items devoted to each topic.) Comparative data, made up of aggregated data from 
all using departments, are available. There Is a provision for departments to write up to an 
additional 5 > multiple-choice Items, which are scored and reported along with the standardized test 
scores. The pricr for the Major Field Achievement Test In Political Science is SI 4 per student 
tested. 



A.l. As a department chair, what Is your overall reaction to this approach to major field 
assessment? (check only one.) 

Excellent Poor 



Very Good Very Poor 

Neutral 



A.2. How likely would your department be to use this test If you were going to assess majors In 
political science? (check only one) 

Definitely will or now using it Probably would not use. 

Probably would use. Definitely will not use It. 

Might or might not use. 

A.2a. Why? 



A.3. What Is the principal advantage of Alternative 1? (CHECK ONLY ONE ) Time Required 

Cost Content Coverage Additional Item Option 

Full Comparative Data Other 

A. 4. What Is the major limitation of Alternative 1? (CHECK ONLY ONE) Time Required 

Cost Content Coverage Lack of Additional Content Modules 

Other 



A.5. What would you like to see changed or added to Alternative 1? 



AHtnntfn.3. MliPr T\tl' "est In BfllHiStl Science Reri«ed to be United States Politics 

Tett with Chnlre of Arirfttlan. ? MnduUf, 



This alternative would use the questions from the Item bank tod from the current Major Field Test 
in Political Science to create a 90-miuute, 90 multiple-choice Item major field test in United States 
Politics. It would be made available as as alternative to the current Major Field Test (at describe 
In Alternative 1). The following modules would he available to supplement the United States 
Politics Major Field Test: 1) Comparative Political Science (30-mlnute module); 2) Political Theory 
and History of Political Thought (30-mlnute module); 3)Internatlonal Relations (15-mlnute 
module); and, 4) Methodology (15-minute module). Departments choosing to use the 90-mtnut* 
United States Politics test would choose either one 30-mlnute module or two 15-minute modules 
to be administered within the two hour time period for the S14 price. Additional modules can he 
given at an additional charge of $3 per thirty mfoutes of additional testing time. A total United 
States Politics score would be reported plus an Assessment Indicators (see Cover Sheet) for each 
module selected. Limited comparative data would be available on the total score for the first year, 
with mr;e complete data becoming available after subsequent nse. 

C.l. As a department chair, what is your overall reaction to this approach to major field 
assessment? (check only ONE) 

____ Excellent Poor 



Very Good Very Poor 

Neutral 



C.2. How likely would your department be to use this test if you were going to assess majors in 
political science? (check only one) 

Definitely will use It Probably would not use. 

Probably would use. Definitely would not use It. 

Might or might not nse. 



C.2a. Why? 



C.3. Which additional modules would your department be likely to use? (check up to three) 

Comparative Politics International Relations 

Political Theory and Methodology 

History cf Polltiril 

Thought None 

C.4. What ts the principal advantage of Alternative 3? (CHECK ONLY ONE ) Time required 

Cost Focus on U.S. politics Coverage of modules available 

Choice of modules Other _ 



C.5. What Is the major limitation of Alternative 3? (check ONLY ONE) Time required 

Cost Focus on U. S. politics Coverage of modules available 

Lack of full comparative data Other 

C.6. What would you like to see changed or added to this approach? 



AJttrnMfvg 2. Current Milor field Political M*nc* T-t with r> H i«n,i M»«tuu« Aw Bttafr|t 

This alternative would augment the present Major Field Test In Political Science (as described in 
Alternative 1) with a choice of three 30-minute modules In tu barest oi United States politics. Each 
module would be composed of 30 multiple-choice questions. The modules, drawn from the item 
bank materials, would be: 1) Institutions, Policies, and Processes; 2) Political Behavior; and 3) 
Constitutional Law/ Judicial Process. Departments could choose to add up to two of these modules 
to the current test. A separate Assessment Indicator score (see Cover Sheet) would be reported for 
each module administered. There would be fee of $2 per module per student in addition to the base 
price of $14 per student tested. Only limited comparative data would be available for the modules 
for the first year with more complete data becoming available after subsequent use. 

B.l. As a department chair, what Is your overall reaction to this approach to major field 
assessment? (check only one) 

Excellent poo r 

Very Good Very Poor 

Neutral 



B.2. How likely would your department be to use an optional module if you were going to assess 
majors In political science? (check only one) 

Definitely will use It. Probably would not use. 

Probably would use. Definitely - >uld not use It. 

Might or might not use. 



B.2a. Why? 



B.3. Which of the additional modules would your department most likely use? (CHECK up to two) 

Institutions, Policies, Constitutional Law/ 

and Process " Judicial Process 

Political Behavior None of these 

B.4. What is the priocipal advantage off Alternative 2? (CHECK ONLY ONE) Time Required 

Cost Additional Content Coverage Choice of Modules 

Other ^^^^^^ 

B.5. What Is the major limitation of Alternative 2? (CHECK ONLY ONE) Time Required 

Cost Extent of Modules Choice of Modules Available 

Limited Comparative Data Other 



B.6. 



What would you like to see changed or added to Alternative 2? 



£» Comparison of Alternatives: 

D.l . Please rank the three alternative approaches to major field assessment in order of perceived 
usefulness to your department (1 highest). 

The current Major Field Achievement Test in Political Science (Alt. 1.) 



The current Major Field Achievement Test In Political Science with three optional 

modules available (Alt. 2.) 

A Major Field Achievement Test In United States Politics (90 minutes) plus a choice 

of modules to test another are* or two (30 minutes), plus the option for the 
department to add other testing modules. (Alt. 3) 

D.2. If you needed to assess your majors, which alternative would you select (CHECK only one): 

The current Major Field Test In Political Science at $14. (Alternative 1.) 

The current Major Field Test tn Political Science plus one of the optional modules 

In Alternative 2 at $14 plus $2 for the additional module. (Alternative 2.) 

The current Major Field Test Sn Political Science plus two of the optional modules 

In Alternative 2 at $14 plus $4 for two additional modules. (Alternative 2) 

The United States Politics Major Field Test with one 30- minute or two 15-minute 

modules at $14. (Alternative 3) 

The United States Politics Major Field Test with one 30 or two 15-minute modules 

plus one or more additional modules at $3 per module added. (Alternative 3) 

VVe would develop our own end-of -major test. 

We would use some other assessment approach. Please describe 



D.3. Do you have any other suggestions for Improving major field assessment in Political Science? 




0 
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E. Background Put*. 

We would appreciate your taking another moment to complete the following background 
questions. 

£.1 . How many undergraduate majors In Political Science did your department graduate last year 

(1988-89)? (CHECK ONLY ONE) 

None 11-15 26-50 

1-5 16-20 51-100 

6-10 21-25 More thin 100 



£.2. What areas can undergraduate Political Science majors specialise in at your institution? 

(CHECK ALL THAT ABE APPROPRIATE.) 

American International Comparative 

Government Relations Politics 

Political Public Public Law 

Theory Administration 

Public Policy Quantitative/Research Methods 

Other (Please specif y.) 

Other (Please specify.) 



Our majors do not specialise 

£.3 What Is the highest degree awarded In Political Science at your institution? (check only 

ONE) 

Bachelors Degree Masters Degree Doctorate Degree 



£.4. Is your department currently engaged In the formal assessment of undergraduate Political 
Science majors? 

Yes (Please answer question E.5.) No ( Please skip to question E.6.) 



£.5. How are you messing your undergraduate majors? (check all that are appropriate.) 

A department-developed multiple-choice examination 

A department-developed essay examination 

A department-developed examination with both multiple-choice nod essay questions 

The Major Field Achievement Test In Political Science 

The GRE Subject Test In Political Science 

A senior thesis or project. 

Other (Please specify.) 

£.6. Do you think your department will begin assessment of undergraduate majors lo Political 
Science within the next two years? 

Yes No Uncertain 



3 Please return this questionnaire In the enclosed pre-addressed, postage- paid envelope. Thank you. 
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Appendix C 

AMERICAN 
POLITICAL SCIENCE 

ASSOCIATION FOUNDED 1903 

1527 NEW HAMPSHIRE AVENUE, NW, WASHINGTON, DC. 20036 • TEL (202)483-2512 • FAX {202} 462-7849 



December 21, 1989 



Dear Colleague: 

The Educational Testing Service is trying to determine whether or 
not political science faculty would find a customized form of the Major 
Field Achievement Test in political science a useful tool for assessing 
what undergraduate majors have learned. To this end, you are receiving 
a questionnaire to elicit your department's potential interest in this 
service. A description of the Major Field Achievement Test is also en- 
closed. 

In the past few years, the Association has received requests from 

• . several departments for assistance in identifying instruments to as- 
sess the achievements of undergraduate political science majors. A 
number of states are mandating some form of assessment. In response, 
to these inquiries, APSA has cooperated with ETS in an effort to de- 
velop an assessment tool that allows faculty discretion over the se- 
lection of test items and/or topics. Now we are asking you to indi- 
cate whether any of the alternative tests proposed in this survey 
might prove useful to you. 

Sincerely, 

Catherine E. Rudder 
Executive Director 



Enclosures 




JUDITH N. SHKLAR, President. Harvard University M. CRAWFORD YOUNG, Secretary. University of Wisconsin 

THEODORE J. LOWI, Presidentelect, Cornell University DAVID BRADY, Treasurer, Stanford University 

ROBERT H. BATES, Vice President Duke University JANE MANSBR1DGE, Program Chair. Northwestern University 

NANNERL KEOHANE. Vice President. wefiesley College SAMUEL C RATTERSON, Managing Editor. APS*. Ohio State University 

^ MICHAEL PRESTON, Vice President University of Southern California (j CATHERINE E. RUDDER, Executive Director 



APPENDIX E 

OTHER UNDERGRADUATE POLITICAL SCIENCE SPECIALITIES 



E2. What areas can undergraduate Political Science majors specialize in at your institution? Other (1) 

Political Psychology Behavior (4) 

BA in International Relations (3) 

PreLaw (2) 

BA - general (2) 

Criminal Justice (2) 

Specialization is not encouraged (2) 

Urban Politics (2) 

American & Public Policy - PA combined, as are IR Corcp. (2) 

Political Science History 

Except for Special IR Concentration 

Legal research assistant 

Not required 

Environmental Politics/Political Economy 

BS in Policy Analysis 

Professional Political Science 

General t International Relations & Public Admin* 

Must take General Course 

Constitutional Law/Judicial Process 

Two tracks: Poll Sci/Public Policy Admin. 

Do not specialize (2) 

Political Science - Business 

Campaign management 

BS - Associated requirements of 12 hours in research methods 

State and Local Government 

General Major without specialization 

f>i 

no 



er|c # # # 



APPENDIX F 

OTHER FORMS OF ASSESSING UNDERGRADUATE MAJORS 



How are you assessing you undergraduate majors? (Other) 

Senior seminar/alumni questionnaire (6) 
Exit interview/surveys (2) 

Research papers throughout curriculum; internship reports; projects (2) 

Grade point averages and fulfillment of major requirement* (2) 

Sampling with essay exams by Jr., in upper level courses 

Externally evaluated 

Multiple Choice exam plus essay 

Self- study process for SACS 

Summary evaluation 

Writing portfolios 

Department developed oral exam 

Will institute 100 item (from subareas) this spring. 
ACAT test developed by Austin Peay State U. 
Assessment being developed for 1992. 

Seminar plus major paper/case study requirements for all majors 
State collaborative test 
Flacement of Graduates 
Will probably use MFAT 

One person department. "I assess them in class every damn day." 
Office of Institutional Research. 
But we'll quit GRE next year. 

I cannot answer these questions (E4-E6) satisfactorily, which demonstrates the problem with multiple 
choice. We require 2 senior seminars as a way of assessing performance in the major. But we do 
not formally college and compare our students performances. 

We are beginning to develop our own assessment. 

Survey of graduates, activities and attitudes. 

I would really like to assess this test with you personally! (Dr. Jo Formicdu, Seton Hall U.) 
Don't know yet, arguing about it. 

Frequency with which they enter law school or graduate school. 

University devised own test. 

Pre- test and post- test in various courses. 

By faculty. 

Outside visiting consultant. 



APPENDIX G 

OTHER PRINCIPAL ADVANTAGES OF ALTERNATIVES 

A3. What is the principal advantage of Alternative 1? (Other) 

No advantage (2) 

Cost and forces purpose 

Force administration to add faculty to department 
Breadth of coverage 
Ve don't have to do it. 

Reaches all students regardless of their specialization 
To satisfy Middle State regarding "Outputs" 
Used a similar one last year 



B4, What is the principal advantage of Alternative 2? Other 
No advantage (5) 

Legitimate education to government assessment 
Depth of testing 

Ve would not have to worry about additional items 



C4, What is the principal advantage of Alternative 3? Other 
Flexibility (2) 

Legitimate education to government 
No advantage 

Difficulty in administering 
Too limited subject area 

K4 
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APPENDIX H 
OTHER MAJOR LIMITATIONS OF ALTERNATIVES 



A4. What is the major limitation of Alternative 1? (Cther) 



Multiple Choice test/not essay (17) 
Methodology (3) 

Would like coverage of public policy analysis and public administration budgeting. (3) 

Fits with departmental objectives (2) 

Difficulty In administering 

Testing Format 

Some good test questions 

No advantage 

Cost/Forces purpose 

Assessment of writing skills; ability to develop arguments 

Revised based on item analysis; some questions poorly written; too heavy emphasis on Western Europe, not 

global enough. 
Weighing of content; item distribution by field 
Difficulty in administering 
Testing Format 
Does not assess skills 
Rigid structure 

Not altogether our percent of breakdown but better than the other two. 
Some modules not relevant. 

Too many applicants are from outside Political Science or from Foreign countries. 
Cost 

The list of schools to which we are comparing ourselves is not terribly impressive. 
Over emphasizes political theory/ though- -under emphasizes IR 
Opposition to national uniform coverage. 

Excessive emphasis on area students in Comprehensive Politics, rather than theory. 
Additional items option good in theory, but would we use it? 
r Can the student think clearly and about politics. 

^ Permits us to test public law/public administration with the 50 question program. 
118 multiple choice questions make topics selected for re-elective. 




APPENDIX H (cont'd) 
OTHER MAJOR LIMITATIONS OF ALTERNATIVES 



B5. What is the major limitation of Alternative 2? Other 

Multiple choice format (9) 
Focus on U.S. (7) 
Methodology (2) 
No advantage 

Need to use Alternative 1 
Still a standard test 

Time, cost, extent of modules, choice of modules available, limited comparative data, 
Difficult to administer 
Poor way to test 

Modules add little that would be of value 

Too extensive 

Non- customized structure 

Written essays by subf ield 

Prefer Alternative 1 - more general 

Too standardized to assess real learning 

It doesn't deal with the basic problem of Alternative 1, 

National uniform coverage 

Basic premise 

Simply would not need it. 

No flexibility 




APPENDIX H (cont'd) 
OTHER MAJOR LIMITATIONS OF ALTERNATIVES 



C5. What is the major limitation of Alternative 3? Other 

Multiple Choice format (3) 

Methodology 

Ethrocentric bias 

More cover of I.R. 

Still standardized 

Not suited for our program 

Change frcus in U.S. Politics to include other 
Choice of questions especially in comparative 
Poor way to test 

Inflexibility in terms of tailoring to student programs 
Module on Constitutional Law/Judicial Process 
National Uniform coverage 

I do not see why I.R, and Methodology have only 15 minutes. 



er|c # 
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APPENDIX I 
PROPOSED CHANGES TO ALTERNATIVES 



A5. What would you like to see change or added to Alternative I? 

Reduce Time/Cost per test (9) 
Needs Essay Questions (6) 
More I.H. questions (6) 

Other tests--e.g., Public Administration (5) 
Make optional ten methodology section (4) 
More balances of items between fields (4) 
Constitutional Law/Judicial Process (4) 

More sub- field coverage- -for example: Public Law/Administration (4) 
Delete Methodology which is not comprehended by the curriculum (3) 
Reduce Methodology (3) 
Non-Multiple choice items (3) 
Item Bank (2) 

Allow flexibility on testing in areas a department cannot cover (2) 
Assessment indicators in Public Administration and Public Law (2) 
Different balance too much U.S. Politics (2) 

Too heavy on History of Political Thought and areas we don't teach, such as Latin America. (2) 
We feature urban politics aid public policy and de-emphasize methodology (2) 
Need to see exam. Would discuss with colleagues, we prefer essays. (2) 

More generalized information, wa found the section cn political theory, specifically, too specialized. 



Develop optional modules (that could yield comparative data) in public law (Const. Law, Judicial Process) 
in Public administration. (2) 

The weakness of public administration/public policy on the test is a problem. But the possibility of 
writing up additional questions is helpful. 



More comparative, less history and political thought more methods. 
Module on non-western government and politics? 
Alternative 2 look like a better choice 
111 advised direction to pursue. 
Research design component 

Expression component and current events component. 



(2) 



More department input. 
Need some alternatives 
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APPENDIX I (cont'd) 
PROPOSED CHANGES TO ALTERNATIVES 



Forget it! 

Balance of questions to reflect courses taken by students rather than what the Professors think the 
questions should be. 

The subfields of Alternative 2 are appealing 

Delete methods indicator, decrease comparative and increase theory* 
Have a major field test in International studies. 
No public law* 

No multiple choice exam would be acceptable. We agree totally with Jacques Barzun's comments on "Multiple 
Choice Flunks Out* in the New York Times . Tuesday 10/11/88, 

The list of schools to which we are comparing ourselves is not to impressive. We are currently deciding 
to use this test or to design our own. This short relatively cheap test is the only alternative which 
is attractive to us. 

Education should not be measured by multiple choice. 

Different questions* -either more in line with our program, or in some cases, more precise/accurate. 
Viable choices 

We would prefer Alternative 1 to 2 or 3 f but might use the IPP module of #2, depending on its content. 

U.S. Government and theory should be increased slightly at the expense of the other three categories. 

Allow department essay questions. 

Add state and local government as separate section. 

Eliminate trivial questions 

Choice to select topic areas 

Measure and analytical and creative capability 

Report assessment indicator regardless if fewer than 10 students tested. 
Pre -test for incoming majors. 
Add content modules. 
Alternative questions. 
Public law as subfield 

Add a component on African American Politics 



APPENDIX I (cont'd) 
PROPOSED CHANGES TO ALTERNATIVES 



B6. What would you like Co see changed or added to Alternative 2? 

Modules with a comparative politics/international relations emphasis (8) 
Public Administration (8) 
Add a section on Political Economy (3) 
Format (3) 

Essay test and short answers (2) 
More modules (2) 
I tea Bank 
Change 

Questions that deal with serious issues in the discipline. 
Sub -divide behavior and institutions 

Flexibility on testing in areas a department cannot cover. 

Allow department to add module (s) based on individual student emphasis area, that would have certain 

amount of customization. 
Political Behavior - general topic compared to other two. 
Some essay - writing by students 
Other fields 

Develop additional modules 
Cor tent of modules not clear 

A geography module and/or information about non-western cultures and religions 
Something on formal theory and methodology 
Flexibility 

Eliminate methodology for those of us who don't teach it. 
Policy 

111 advised direction to pursue. 

Comparative data ^ * 

We feature urban politics and public policy and de-emphasize methodology, ■ * 

Senior research paper and/or comprehensive exam, 
( ^ Multiple choice not a good indicator of knowledge. 

Wider range of modules- -beyond US alone, opportunity for local questions. 

Essay answers, if not feasible, possibly hypcthetical situations emphasizing reasoning, analytical skills, 
etc. 

Addition of choice 

Some essays v perhaps similar to the Advanced Placement exams. 
Over emphasis on American Politics. 
Make more like Alternative 3. 
Modules for every area. 

Inclusion of Non-Western and Politics Module. 
Retention of the fifty locally generated questions option. 





APPENDIX I (cont'd) 
PROPOSED CHANGES TO ALTERNATIVES 



What would you like to see changed or addjd to this approach? 

i 

Module on Public Administration (6) 
Add essay questions (5) 
Public Lav modules (3) 

Modules of equal length (in Pol, Theory, American Politics, including constitutional law— without 
additional cost. (2) 

More coverage of IR? Public Law/Judicial Process either U.S. Module (2) 

Cost/Time (2) 

4*15 min. modules 

Offer other type questions 

Would not serve needs 

Focus on theory in each area 

Too narrow 

t ^t students pic 1 modules individually 

ition of 15 min. comparative pol. & 15 min. Political theory, IR 
Module on State & Urban Politics 
International Lav 

Return to inclusion of "modules" into basic test 

Use Am. Politics base but individualize the additional modules to reflect the areas students selected 
for concentration in structuring major. 

Matching international Relations with methodology in cost structure inhibits using only one, make this 
separate . 

Make one vith international studies exam. 
Add more on U.S. government 

We feature urban politics and public policy and de-emphasize methodology. 

Too focused on American Politics 

Broaden 

A National security policy module. 

Multiple choice not good indicator of knowledge. 

It is unfortunate that you have not been able to develop a question bank. 
A political behavior module would be useful to us. 
Opportunity for local questions- -Longer international relations 
Possibility of crossing greater number of modules. 

We object to this narrow approach P.S. You need to cover at least 5 subarea list in Alternative 1. 
Do an I.R. version 

A non-Western Government and Politics module included. 



APPENDIX J 

OTHER SUGGESTIONS FOR IMPROVING MAJOR FIELD ASSESSMENT 



D3. Do you have any other suggestions for improving major field assessment in P.S. 
Need essay (9) 

Offer option of locally generated add-ons (5) 

Cover Publ. Admin. (3) 

Do international studies exam. (3) 

Item Bank idea for customization of assessment instrument. (3) 

Ability to use knowledge of acts learned; the ability to think like a trained political scientist? (2) 
Equal weight to comp. Pol. f methods, IR and Theory (2) 

Scrap the whole idea, leave up to individual univ., stress conceptual and ethical issues. (2) 

Reduce the cost (4) 

Need special exam in Pub. Admin. (2) 

Broaden. (2) 

Certainly not to restrict it to U.S. politics. 
More modules 

Carefully constructed senior seminar to be required of all majors. 

Construct module tests of equal length in each of the fields, let student/depart, choose combinations 
they wish. Our majors choose their own emphasis (3 out of 4 field) from American, Theory, I.R.'s, and 
Comparative . 

Please design an international students test for us to consider. 

As Indicated, we are mostly concerned with the cross- institution comparisons and believe that at present 
the list does not contain sufficient range of quality in institutions. 
Drop the idea it is terrible t 

Find a method that assess the use and expression of knowledge, and provides written response to analytical 
questions. Actually, your very much out-of-date. 

If you are going to a choice format, you need to allow enough choice to cover most cases. These 

alternatives simply do not do that. 

Test with increased IR and an Economic Section. 

We like the possibility of adding some local questions but worry about the national norming of such items. 
Need pre- test and post test to help ascertain how much the P.S. dept. has contributed to the student's 
knowledge . 

No demand for standardized test. 

What is the status of public policy* -a part of U.S. or does it now merit its own module? 
Would only do it if required by administration 
Don't do it this way; why must there be one standard; etc. 
Add a non-Wester Gov't Module 
, Continue to work on Item Bank ^ ^ 

: * i Noc good for Pub. Admin, student 

Depts. should concentrate on improving curriculum, teaching Y exam in courses 

ERIC • • • 



APPENDIX J (cont'd) 
OTHER SUCGESTIONS FOR IMPROVING MAJOR FIELD ASSESSMENT 

(cont'd) 

Be certain that the assumption, goals, and standards underlying each part of the exam are clearly se 
out. 

Ability to assess alternatives, develop arguments and the like. 

Not so much reliance on multiple choice, would like to see some analytical and creative capabilities. 
Better departmental control of assessment 
Public Admin. Module. 

Alternative 2 would be helpful with one of these modules 

Report assessment indicators for each student, even if fewer than 10 students tested. 

Improve questions, too many reply on opinion. 

Reduce emphasis on political economy. 

There is a need for a Pretest and the test listed. 

Make sure subfields (modules) scores are reported. 
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APPENDIX K 

OTHER MEANS OF ASSESSING MAJORS IN POLITICAL SCIENCE 



If you needed to assess your majors, which alternative would you select? 
Essay exam (3) 

Own test or some other assessment would have to think about it. (2) 

Capstone course with reading and paper requirements (2) 

Modules of current test from which we construct test 

Senior Comp exam/with essay question in 4 fields plus objective test. 

Close supervision of independent research. 

Survey of graduates asking them to assess the value of their education in P.S. majors. 
Oral exams 

Current system, course grade and GRE 

Comprehensive exam, course exam, and 3 research papers, qualitative exit interviews & surveys 
Some approach that tests critical thinking skills and problem solving would be our goal. 
Question on I.E. & Corap Politics 

Curriculum taken by major: questionnaire administered to majors 
Combination of alternatives: Test; Senior Seminar and Honors Seminar 
Combination of essays, multiple choice and presentation. 

It would be an essay approach, multiple choice do not elicit real 1 naming. 
Might be in part self developed 

Instructions too complicated and long at ^.tart of exam, this could be more efficient. 

Measure behavioral change of major, comprehensive exams with 1,3 & 5 year follow-up of graduates. 

We will be developing our own test, despite t. e fact that this will not permit cross institutional 

comparisons . 

The major weakness we see in this approach has been the lack of a section on public administration/public 
policy. 

Reading student papers and tests. Talking to the, We know our seniors very well after 4 years. 



APPENDIX L 
REASONS FOR LIKELIHOOD OF USE 

A2 - How likely would your department be to use this test (current Major Field Achievement Test) if you were 
going to assess majors in political science? 
A. 2a. WHY 



Positive 

Content Coverage (14) 

Comparative data (9) 

University requires comprehensive 
test - majors uust pass before 
graduation (4) 

Convenience (4) 

Assessment pressures (3) 

SASC requires (3) 

Simplicity (2) 

Objective - non- college 
Instruments helpful to know 
background of political science. 
(2) 

State requires accountability- 
MFAT under consideration as ppg 
indicator (2) 

Cost factor might prohibit (2) 

Would use if other alternatives 
were not available (2) 
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Test would help us assess 
effectiveness of our 
Instructional program* (2) 



Neutral 



Cost (13) 



Still considering several options 
(5) 

Objective test questions (4) 

Currently exploring alternatives 
(4) 

Many applicants are from outside 
Political Science or from Foreign 
countries- (3) 

Risks of political use of results 
in a state that is assessment* 
crazy. Risks of teaching to the 
test. (2) 

Convenience (2) 

More attention to International 
Relations (2) 

Not sure of institutional 
requirements and funding or of 
alternate tests (2) 

Would need to ~e actual test (2) 

Economy of efforr are in-house 



Negative 

Not appropriate for 
curriculum/content coverage (21) 

Will not use multiple choice exam 
(24) 

Cost/Time (5) 

Too heavy emphasis on theory (2) 

It doesn't measure how to think 
creatively (2) 

Prefer current method (2) 

Too general (2) 

No interest at this time (2) 

Offer own oral/essay exam 

Discipline based test not of use 
Reflects bias of "mainstream 11 of 
discipline 

Other options being considered 

Infringes on Academic freedom 

Little agreement concerning what 
constitutes Political Science 

Will not do it until mandated 
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Convenient and good reputation 

(2) 

Availability (2) 
Would do Job 

Provides barometer for Department 
Teaching effectiveness 

External validity/math comparison 

Need additional information 

Governing Board requires test 
with National Norms 

Useful to assess majors and to 
people who want to exempt course 

Presently using test developed by 
Austin Peay Univ. 

Who Pays 

Familiarity 

No definitive position in the 
department on assessment tests 

If cost is not great 

If we went to an assessment 
program we would need a solid, 
standardized test for comparative 
purposes. 

We are using it (used once) and 
deciding what to do in the future 

Over emphasizes political 
theory/thought • • under 



Some dissatisfaction with some 
questions 

Could meet requirements for 
outcome measure 

Assess level of information of 
majors compared to other schools 

Hard to know quality of questions 
-~ ETS makes copies available 
under overly restrictive 
conditions 

No courses/faculty in political 
theory and coverage of 
methodology is weak 

Needs assessment to identify 
requirement for use 

Institution is pursing for 
assessment instrument 

No need at this time 

Need to compare with ACAT 
currently used 

Created own instruments - murt 
test before we decide 

Content coverage 

Funding by University 

May be required to assess against 
nationally normed test 

Does not include items on Public 
Administration/Public Policy 

Not sure of testing as a major 
method of assessment 



Does not really assess learning 
in more than superficial manner 

Pedagogical philosophy 

No required distribution that 
would be needed 

Biased against international 
relations as a provision U.S. 
Govt. 

Using external examiners; would 
be added burden on students 

Testing shapes curriculum 

Institution does not user 
comprehensive end of major exams 

Small number of students 

Evaluations made course by course 

Not comfortable with * recall of 
content* type of testing as 
outcome measure* 

Prefer essay comprehensive exams 
Need a comparison with national 
norms, but not this one. 

Apples and oranges 

We offer a minor but t*>t a major 
in Political Science 

Faculty may prefer to develop own 
instrument 

I looked at it, questions too 
picky. 



emphasizes International 
relations 

An assessment instrument has been 
suggested for all majors at 
L.O.C. 

Want to test across full range of 
areas 

Flexibility - use of 50 
additional questions from the 
department 

We've been leaders in assessment; 
so we appreciate your willingness 
to work with us. 

Assess the cumulative knowledge 
of the field. 

University is promoting outcomes 
assessment. 



Too much fact and not enough 
ideas 

LSAT preferred 

Not flexible enough 

Need to see exam/test first 

Monitoring and assessment 

College adopted it, so we're in. 

Multiple choice questions are a 
poor substitute for analytical 
essays. 

For the past year, an intense 
effort has be underway to test 
our students with essay 
assignments. We in the Political 
Science departments have been 
doing this for 23 years. 

Depends on results of an outcomes 
committee study. 

Broad in scope; departmental 
freedom. Negatives: too much 
emphasis on comparative and 
methodology 

Political Science, as a major 
field, is growing in enrollments 
at the undergraduate level. Most 
regard it as a public service or 
pre -law program. 

Time 

A somewhat too shallow basis for 
assessment 



Our majors would be interested in 
3 sections: 1R, Comp. Polic, 
and Methodology 

No public law for pre -law majors 
we attract 

Only a few majors don't feel 
intellectual sorting is useful 

The approach defeats the purpose 
of higher education in the 
liberal arts 

We feature urban politics and 
public policy and de- emphasize 
methodology 

No pressure to do so 

Some questions ambiguous 

We used the ETS pilot In year 1. 
Disappointed in content of Items 
(often descriptive/idio- 
syncratic/discrete information 
rather than patterns) and, re: 
coverage, far too heavy on 
normative political theory, far 
too light on international. 
Weaknesses speak to changes we 
prefer. 

We felt the test dii not 
effectively measure or grasp 
concepts, idealogies, reasoning, 
etc* 

We are most probably not going to 
assess major in this way. 

Faculty would not support a major 1 ^ 
field test 
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Fee majors at present 

Not sure this test would be an 

accurate measurement of our 

people 

I don't believe in this process 

The university wants us to assess 
our own majors 

Structure does not fit out 
curriculum 

Along with other criteria; senior 
research paper, comprehensive and 
exam at 4th year* 

Department needs to achieve 
consensus on the justification 
for such a test 

We have developed our own 
evaluation system, fitted to both 
major and liberal arts. 

I reviewed an actual test and it 
was loaded with comparative- -esp. 
Latin American questions. 

Currently we offer little 
comparative theory or 
methodology. 

We would probably require an 
institutional commitment. 

We do not have methodology, this 
would lower ecores. 

Would need to examine impact in 
department. 



Not useful at this time* 

We reviewed an earlier version 
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APPENDIX L (cont'd) 
REASONS FOR LIKELIHOOD OF USE 

How likely would your department be to use an additional optional 
majors in political science? Why? 



Positive 

Flexibility (4) 

Reflects curriculum 
better/coverage (2) 

Convenience (2) 

Constitutional Law & Judicial 
Process requirement part of 
curriculum. (2) 

Constitutional Law & Political 
Behavior 

Our department focuses mainly on 
module* 

Based on Cost, this would be a 
••cheap way* to evaluate our U.S. 
Gov't program* 

Use in classes only 

To be thorough. 

Compared to present test, would 
be helpful* 

Show how strong pre -law majors 
are in constitutional 
law/judicial process 

Too much emphasis on methodology. 

Cover a broader base in our 



Neutral 

Cost/Time (8) 

Constitutional Law attractive for 
Pre -Law, other modules not 
relevant to program (4) 

Would depend on content. (2) 

Need to see actual test. (2) 

Administration skeptical about 
standardized tests. (2) 

Needs assessment to identify 
requirements . (2) 

Do not like multiple choice 
testing (2) 

Modules would better mirror 
program emphasis (2) 

1) Would have to see whether 
questions are well constructed 
and 

2) would prefer range of modules 
beyond U.S. 

Greater strengths in advanced 
areas of U.S. Politics makes this 
generally more appealing. 

Department may feel data not 
useful. 



module if you were going to assess 

Negative 

Too focused on U.S. Politics (17) 

Limited course offerings (11) 

Cost/Time (8) 

Multiple Choice (17) 

Too specialized in sub- 
disciplines (5) 

Offer own oral/essay exam. (4) 

Alternative 1 is sufficient (3) 

Additional fluids not very rseful 
(2) 

No need at this time. (2) 
Overloaded on one area (2) 
Too narrow (2) 

Does not suit our emphasis or 
approach. (2) 

Does not assess learning (2) 
Small number of students (2) 



Our major is International ( 
studies. (2) 

Pedagogical Philosophy 
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discipline and make provisions 
for adding content modules. 

Univ. System requirements 

Institutional pressure. 

If we vent to an assessment 
program, we would need a solid, 
standardized test for comparative 
purposes . 

A majority of our undergraduates 
in P.S. majors regard the program 
as a preparation for domestic 
public services or pre -law 
school. The faculty regards it 
as a means to enhance capability 
of an attentive public. 

To obtain additional information 
upon which to base on evaluation 
of individual students. 




Comparative politics of equal 
importance in our curriculum, 
would be under -represented, 

Heavy stress on U.S. Politics. 

No consensus among department 
faculty as to value and cost 
effectiveness of such tests. 

Few majors at present. 

I don't believe i this process. 

Coverage in option 1 probably 
adequate . 

Department needs to achieve 
consensus on the justification 
for such a test. 

Should be essay style. 

Flexibility 

We have a general is ts major, 
about 36 semester hours. 

There must be an outlet to allow 
questions pertinent to the 
teaching in a specific 
department. 

While this approach may be valid 
for some, I don't fee that the 
three areas listed are 
particularly under- represented. 

Risks a political use of results 
in a state that is assessment 
crazy. Risks of teaching to the 
test. 




Modules not central to major. 

Seems too specific for a 
baccalaureate program. 

We don't require all the courses. 

This approach defeats the purpose 
of Higher Education in the 
Liberal Arts. 

We feature Urban Politics and 
public policy and d-*- emphasize 
methodology. 

No pressure to do so. 

Not likely to improve much on 
Alternative 1. 

We are looking for a 
comprehensive, broad exam. 

Modules don't mach our 
specifications. 

Do not plan on using Major Field 
test. 

We could cover the material in 
Option 1 modules, and move in the 
"additional" 50 items in 
Alternative 1. 

We are probably not going to 
assess majors in this way. 

Not flexible enough. 

Lack of interest within *V r n 
department. 




Would need to examine impact in 
department. 

Would use in the absence of 
something better. 

Specific modules may not be 
appropriate for our students. 

Not at the stage to determine our 
exact needs. 

Would provide a goal or target 
for seniors. 

Must see exam/test first . 

Monitoring and assessment 

If students could choose their 
own optional modules. 

This is better than the first 
one, but have developed our own 
instrument based on the courses 
we offer in our program. 

I'm not sure what the content of 
the modules would be compared 
with the MFAT. 

N^t thrilled with base 

Public Policy, Public 
Administration 

Eliminate trivial questions. 

Not broad enough 

Pattern of courses taught in [ | 
department. 




Don't like multiple choice tests. 

Curriculum is more accurately 
represented in the current MFAT. 

Not needed for assessment 
requirements 

Testing shapes curriculum. 

Institution does not use 
comprehensive end of major exams. 

Too particularized for our 
undergraduate curriculum. 

Evaluations made course by course 

Don't like the conceptual 
organization 

I looked at it, questions to 
picky. 

Not comfortable with "recall of 
content* testing 

Apples and oranges 

No major in Political Science. 

Not well suited to the diverse 
structure possible in the major 
programs. Items seemed 
inappropriate for general 
assessment. 

Our students emphasis 

Discipline based test not of much 
use . 

Do not meet our needs 



Depends on whether module (s) can 
be added on individual basis to 
fit each student's emphasis area, 
rather than blanket coverage ♦ 

Don't have masters, Ph.D. program 

Need measure of accrediting 
process 

Variety of sub -areas- -may be 
difficult to administer. 

Creating own instrument 



Reflects bias of the mainstream 
of discipline 

Public Administration 

Prefer current method 

Faculty may prefer to develop own 
Instrument. 



Department too small for this 
degree of specificity 

Unclear of content. 

Need to compare with other 
methods of assessment. 

Developed own evaluation system. 

Not all of our students take 
courses in each of the modules. 



APPENDIX L (cont'd) 
REASONS FOR LIKELIHOOD OF USE 



C2a. How likely would your department be to use this test (U.S. Politics 
you were going; to assess majors in political science? Why 



Positive 
T .>cus on U.S. Politics (6) 
Best suits our needs (2) 
Flexible (2) 

Reasonably assess areas of 
student (2) 

Coverage (2) 

Allows students to demonstrate an 
area of specialty. 

It would provide feedback to 
instructors concerning material 
covered or omitted. 

Good base and modules. 

Too Broad 

If we were to drop state test 
Because methodology appear to be 
eliminated 

University system policy 
Too Narrow 
Comprehensive 

Convenience and good reputation 



Neutral 

Cost/Time (6) 

Too much emphasis on U.S. 
Politics (2) 

No modules on State & Local, 
Public Admin. , Public Policy. (2) 

Allows test to be tailored to 
what dept. offers , but maintains 
level of generality. (2) 

Focus of review is student 
satisfaction currently. 

Has the flexibility to match 
curriculum better than the other 
options. 

Under emphasis on data I.R. 

No provision for local questions. 

No provision of U.S. politics 
into modules as in Alt. 2 

Would need to examine impact in 
dept. 

More flexible that Alter. 1. The 
full package would be more 
comprehensive than Alter. 1. 



focus with additional maodules) if 



Negative 
Too focused on U.S. (47) 
Multiple Choice (10) 
Length (4) 
Cost (4) 

No need at this time (3) 

Too little Int. Rel. emphasis (3) 

Structure does not fit our 
curriculum (2) 

Offer own oral/essay exam (2) 

Students required to take courses 
across the discipline. (2) 

Biased to standardized test. (2) 

Test Format (2) 

Prefer current method (2) 

To complex (2) 

Too particularized (2) 

Inferior to Alternative 1. (2) 

Not certain yet if we would use a 
standardized test or construct 
our own, (2) 



Broader ability to fit test to 
curriculum . 
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Would it have equal validity? 
Could better represent offerings 
in our limited de^t. 
1 want what the students have 
studied FACT. 

Multiple choice questions are a 
poor substitute for analytical 
essays. 

Flexibility 

1 still favor using an internally 
developed test, but an outside 
instrument in addition might be 
useful. 

No assessment as to value of such 
tests. 

Fee majors at present. 

Need to see actual test. 

We don't like the choice of 
modules . 

If we go to standardized test, we 
would use them in survey course 
and precede from there. 

Department needs to achieve 
consensus for the Justification 
for such at test. 

We are eliminating course 
concentrations next year to 
create a broader understanding of 
government. 

Might use without additional 
modules . 




Reflects bias 

Major program is comprehensive, 
not just U.S. Politics plus add- 
ons. 

Ethnocentric 

Pedagogical Philosophy 

To include IR must pair with 
methodology or add to basic test. 

Testing shapes curriculum. 

Like general approach better 

Focus too narrow. A substantial 
minority of our undergraduate 
majors have need fot assessment 
in areas covered by Alternative 
1. 

Does not access Learning 
Objective test 

We prefer a short and general 
exam. 

This approach defeats the purpose 
of Higher Education in the 
Liberal Arts. 

We feature urban politics £ 
public policy and de~emphasi * 
methodology , 

We would want the benefit of, 

coverage in Alternative 1. ff) 

Current plan with optional module 
seems preferable. 



Too Specialized 

Need assessment to identify 
reason for use. 
Consistent with curriculum 

Need a measure 

Institutional requirements and 
funding 

Creating own instrument 

Not sure of testing as a method 
for assessment 

Must see test/exam first 

Monitoring and Assessment 

Test needs to be essay 

Do not use Standardized test 

Alternative 1 preferred. 



Cast of coverage, though I like 
the depth and ability to 
construct our own exam* 

More comparative data* 

Institution does not use 
comprehensive end of major exams. 

Small number of students 

Our emphasis is cress -cultural 

Don't like weighing. 

Not comfortatle with "recall of 
content" testing 

Opposition in department 

Apples and oranges 

Proposed format still to rigid . 
Use of Amer. Politics as base is 
not bad, but modules should vary 
with student program of study* 
Our students do not proceed in a 
look step fashion. 

Admissions officer handles this. 

I looked at it. questions too 
picky. 

Too limited for a general 
baccalaureate program* 

Might use expanded fmpplements 

Unbalanced for our curriculum. 
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